QualityCrowd - A framework for crowd-based quality evaluation
نویسندگان
چکیده
Video quality assessment with subjective testing is both time consuming and expensive. An interesting new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the internet. We therefore propose in this contribution the QualityCrowd framework to effortlessly perform subjective quality assessment with crowdsourcing. QualityCrowd allows codec independent quality assessment with a simple web interface, usable with common web browsers. We compared the results from an online subjective test using this framework with the results from a test in a standardized environment. This comparison shows that QualityCrowd delivers equivalent results within the acceptable inter-lab correlation. While we only consider video quality in this contribution, QualityCrowd can also be used for multimodal quality assessment.
منابع مشابه
A Data-Driven Framework for Visual Crowd Analysis
We present a novel approach for analyzing the quality of multi-agent crowd simulation algorithms. Our approach is data-driven, taking as input a set of user-defined metrics and reference training data, either synthetic or from video footage of real crowds. Given a simulation, we formulate the crowd analysis problem as an anomaly detection problem and exploit state-of-the-art outlier detection a...
متن کاملToward a Robust Crowd-labeling Framework using Expert Evaluation and Pairwise Comparison
Crowd-labeling emerged from the need to label large-scale and complex data, a tedious, expensive, and time-consuming task. One of the main challenges in the crowd-labeling task is to control for or determine in advance the proportion of low-quality/malicious labelers. If that proportion grows too high, there is often a phase transition leading to a steep, non-linear drop in labeling accuracy as...
متن کاملCrowdsourced Top-k Algorithms: An Experimental Evaluation
Crowdsourced top-k computation has attracted significant attention recently, thanks to emerging crowdsourcing platforms, e.g., Amazon Mechanical Turk and CrowdFlower. Crowdsourced top-k algorithms ask the crowd to compare the objects and infer the top-k objects based on the crowdsourced comparison results. The crowd may return incorrect answers, but traditional top-k algorithms cannot tolerate ...
متن کاملPair Me Up: A Web Framework for Crowd-Sourced Spoken Dialogue Collection
We describe and analyze a new web-based spoken dialogue data collection framework. The framework enables the capture of conversational speech from two remote users who converse with each other and play a dialogue game entirely through their web browsers. We report on the substantial improvements in the speed and cost of data capture we have observed with this crowd-sourced paradigm. We also ana...
متن کاملA Hyper-Heuristic Framework for Agent-Based Crowd Modeling and Simulation: (Extended Abstract)
This paper proposes a hyper-heuristic crowd modeling framework to generate realistic crowd dynamics that can match video data. In the proposed framework, motions of agents are driven by a high-level heuristic (HH) which intelligently selects way-points for agents based on the current situations. Three low-level heuristics are defined and used as building blocks of the HH. Based on the newly def...
متن کامل